Audio-visual musical instrument recognition

In 2008, a humanoid robot developed by Honda conducted a symphonic orchestra in front of a live audience. It perfectly imitated the pre-recorded actions of the orchestra’s human conductor. Despite its realistic movements, it was suggested that the robot could not listen to nor interact with the orchestra as a true conductor would. For a robot to truly direct an orchestra, it would need to hear, distinguish, and respond to the sounds of different instruments. As a first step to making a musically trained robot, we implement monophonic instrument recognition on Hearbo, developed by HRI-JP for audio-based human-robot interaction. Until now, musical instrument recognition has been limited to audio recording analysis. In the field of solo musical instrument recognition, acoustic features such as MFCCs and LPCC’s [1] have been widely studied. Classifiers like k-nearest neighbors [1], SVM and GMM [2] to classify these features have also been examined. Using a priori musical knowledge, Martin’s classification system in [3] used a hierarchy of musical instrument classes, such as grouping string instruments like guitar and violin together. Both his and Klapuri and Eronen’s [4] work showed better results classifying at the instrument family level rather than at the specific instrument. Brown [5] investigated the use of features such as attack and decay to distinguish between four similar woodwind instruments. Indeed, a common problem comes down to distinguishing instruments of the same family. So far, no one has yet exploited the visual differences between instruments to overcome this problem.